1) The program requires Hypercard Player, or Hypercard 1.2.5 or above;
2) Check that you have the following 22 files: P1-P10, STARTUP, documentation, neuralnetworks, simulator, numbers, num1, graph1, graph2, tut1.1, simul.main, TreffNet.ReadMe, TreffNet.Documentation;
3) Load all files into a single folder called TreffNet on the hard disk of your Macintosh;
TreffNet is a simple, Hypercard-based introduction to learning in neural networks. It consists of a tutorial on the simplest of network learning methods called Hebbian learning. The stacks consist of (1) tutorial animations, and (2) a graphical network simulator that can learn to associate two pattern vectors. Based on the first chapter of the classic text by Rumelhart and McClelland (1986), it "brings to life" the basic concepts and mathematics that need to be fully understood before the student tackles more complex systems.
TreffNet was designed to be distributed as Shareware. It is the hope of the author that if you like the package and find it useful in your own learning and teaching others about neural nets, then you send a small Shareware fee of $15 to the author. Alternatively, you can send me a CD of music from your part of the World! Receipt of your donation will result in me sending you a non-Shareware version of the program. TreffNet cannot be sold or distributed for profit without the author's written consent. However, distribution via online services is acceptable provided that all original components and unaltered files are included in the package.
Treffnet is supplied as is. All responsibilities lie in the user, not the author. The person using TreffNet bears all risk as to its quality and performance. No responsibility is taken by the author as to the program's consequences, neither explicit nor implied.
This program explains some of the major concepts needed to understand learning and representation in models of nervous systems. In recent years, the field of psychology has experienced a renaissance of interest in the biological and physical basis of memory, learning, perception and action. This has partly been due to the power which computer modeling has provided researchers in simulating our own and other animals' behavior (e.g., Anderson & Hinton, 1981).
The last decade has also seen the development of elegant and powerful artificial neural-network mechanisms which simulate some of the details of real nervous systems (McClelland & Rumelhart, 1986; Rumelhart & McClelland, 1986). The recent excitement about neural networks is due to the hypothesis that mental states (e.g. memories, dreams and perceptions) are a kind of "emergent property" of the many simple processes at the microscopic scale of continuous neural activity. Neural networks show that "knowledge" may not exist in the brain as simple "pictures", words or symbols (as the approach of symbolic AI assumed in the 1970's), but instead may be coded as a diffuse, distributed pattern of activity distributed across large ensembles of neurons. Neural network models allow the simulation of certain facts from neuropsychology and indicate that “thoughts” and “percepts” are not constructed and represented in any single, local part of the brain, but are instead distributed and superimposed over large areas of nervous tissue in a pattern of electrical and chemical activity. In particular, because different patterns are all stored in the same nervous tissue, very interesting effects can naturally occur such as the mixing and blending of one pattern with another to creating new patterns and generalizations.
In a connectionist model, knowledge is embodied or stored as a simultaneous pattern of neural strengths in the multiple connections between neurons. This approach to cognitive science has therefore become known as "Parallel Distributed Processing" (PDP) (McClelland & Rumelhart, 1986), but is just as often called connectionist or network modeling, for obvious reasons.
Although such a constructivist approach toward explaining mental states may in fact be fundamentally flawed due to an emphasis on the organism alone with no serious investigation of the impact of information from the environment (Treffner, 1987), neural networks do provide a powerful mechanism in which to examine the material reductionism in the representationist hypothesis that mental states may be brain states.
Implications of connectionism
Connectionist modeling has been used to help understand the mechanisms upon which human behavior are founded. It has also been used to support the suggestion that animals and humans may be more similar psychologically than previously realized. We can learn about ourselves through studying the neural processes of other organisms and then showing that both depend upon similar principles that can be modelled. The philosopher Descartes, and those who followed him, believed that non-human species were simple, unconscious machines but that we were different since we possessed "consciousness". The new models of neural networks show how the complex behavior of both humans and animals might be conceived in the same physical terms—as the result of a miriad of interactions occurring in their nervous systems. Old issues such as the existence of animal consciousness can then be given a fresh interpretation since we can ask questions guided by the similarities between neural facts, predictions from network models, and observed behavior.
Although highly successful, the neural network approach is not without its critics. For example, the central problem of how to discover or define appropriate features on the input units (the problem of information!) has never been adequately addressed by network representationist approaches (Treffner, 1987). One answer to this fundamental issue comes from a realisation of the the deep evolutionary and ecological constraints that must apply to any purported mechanism underlying the dynamics of perception and action. From this perspective, network architectures would need to conform to the level of dynamical and ecological constraint, a level of constraint usually ignored in network modelling (Gibson, 1979; Kelso, 1995). It is perhaps with respect to this informational and ecological level of analysis that network models might still provide useful insights into the mechanisms supporting the behavior of complex systems (Treffner, 1987, 1997).
It is hoped that this tutorial will provide the student with a concrete grasp of the basic hypothesis of connectionist representation: how many very simple and meaningless processes at a microscopic neural scale might provide a substrate on which macroscopic ensemble effects emerge at the scale of everyday experience.
REFERENCES
Anderson, J. A., & Hinton, G. E. (1981/1989). Parallel models of associative memory. Lawrence Earlbaum Associates. (The volume which set the ball rolling for the PDP revival of the 1980's.)
Caudill, M., & Butler, C. (1990). Naturally intelligent systems. MIT Press. (A readable introduction to the main ideas of connectionism. No equations, no references, just English!)
Gibson, J. J. (1979/1986). The ecological approach to visual perception. LEA.
Gonzales, M. E. Q., Treffner, P. J., & French, T. (1988). A naturalistic approach to mental representation. Presented at the Int. Conf. on Thinking, Aberdeen. Reprinted in Gilhooly, K. J., Keane, M. T. G., Logie, R. H., & Erdos, G. (1990), Lines of thinking: Reflections on the psychology of thought, Wiley. (Reconsiders the debate between A. I. and PDP from the standpoint of ecological psychology.)
Kelso, J. A. S. (1995). Dynamic patterns: The self-organisation of brain and behavior. MIT Press.
McClelland, J. L., & Rumelhart, D. E. (1986). Parallel distributed processing, Vol. 2: Psychological and biological models. MIT Press.
&
Rumelhart, D. E., & McClelland, J. L. (1986). Parallel distributed processing, Vol. 1: Foundations. MIT Press. (These two researchers originated the term "PDP" and their volumes have become the essential references in the area. The necessary equations are explained in sufficiently transparent detail.)
Rumelhart, D. E., & McClelland, J. L. (1988). Explorations in parallel distributed processing. (A companion volume to PDP 1 & 2. Tutorial format which includes programs for simulating the many different network models available. Additional explanation of the equations. Inexpensive.)
Treffner, P. J. (1987). Ecological connectionism and animal-environment mutuality. In M. Caudill (Ed.) Proc. IEEE 1st Int. Conf. on Neural Networks, Vol. 2, pp. 813-820, San Diego.
Treffner, P. J. (1997). Representation and specification in the dynamics of cognition: Review of Robert F. Port and Tomothy van Gelder (Eds.), Mind as motion: Explorations in the dynamics of cognition. Contemporary Psychology, 42, 697-699.
The program is divided into two parts. The first part consists of Menu Options 2 to 9 and provides a Tutorial on neural network models of associative memory. Starting with an outline of what a single neuron is and does, the user can see how to build a network which implements a simple pattern associator using a form of synaptic weight adjustment called "Hebbian learning". It is strongly recommended that the user works through all these tutorials in sequential order. Just click on the topic in the Menu screen.
The user can view screens as fast as he or she wishes; the tutorials are, for the most part, self-paced. This means the user can click on a NEXT button at the bottom of the screen to view the next screen in a given tutorial. You can return to the menu at any stage in a tutorial by clicking the MENU button in the bottom corner of the screen.
TO STOP PROGRAM EXECUTION: If at any stage the user wishes to stop program execution, then the "Apple/command” key should be held down while simultaneously hitting the “.” (period) key.
The Simulator
Menu Option 10 takes the user to the other major part of the program which is a neural network Simulator itself. The simulator allows the user to watch the network learn a Hebbian association between two patterns of neural activity, and subsequently test the new pattern of connection weights by presenting incomplete or partial patterns. This demonstrates the process of so-called "pattern completion" which is thought by some researchers to occur in the brain, for example, when seeing parts of objects that are occluded by other objects. In this case we know (or perceive) that a whole object is really there rather than only the part visible (e.g. we know or perceive that a person occluded by a table isn’t really half a person!). The simulator network is based on the simple introductory network presented pp. 34, 35 of Rumelhart and McClelland (1986). Readers might find it useful to refer back to that chapter while using the simulator.
A second aspect of the simulator demonstrates how five digits (e.g., 1, 2, 3, 4, 5) may each be represented as a pattern of activation and then learned and stored in the same network. This can be considered a "real world" example of pattern recognition and pattern completion. In addition, the numbers simulation also allows the user to build NEW associative networks of ANY NUMBER of input and output units, thus permitting the learning of any association required.
The simulator allows the user to experiment with the Hebbian learning process by entering pairs of patterns to be associated together and then watching how the connection strengths change. The best way to become acquainted with the simulator is to click SIMULATOR INTRODUCTION in the SIMULATOR MENU. This eventually takes you to the simulator itself. You can then follow the instructions given on how to enter the input and target patterns to be learned.
Be sure to read below what the function of the various buttons, fields, and screens are before clicking on SIMULATOR INTRODUCTION.
GRAPHICS vs. MATRIX DISPLAYS:
You have the option of either of two display formats:
1) GRAPHICS: a "classical" network-type display with the input units below fully connected to the output units above them, or
2) MATRIX: a "matrix" display which allows a more transparent picture of the pattern of individual weights in terms of positive and negative values (i.e. the "weight matrix")
The advantage of the GRAPHICS mode is that it uses a display format analogous to that previously used in the introductory tutorials.
The advantage of the MATRIX display is that in addition to the clear display of weight patterns, this format is also used in the NUMBER LEARNING NETWORK example which requires 15 input and 15 output/target units. This means there are a total of 225 weights which can be changed each cycle which is far too many weights to display using the GRAPHICS type format.
THE SETUP SCREEN
This screen allows the user to enter the main parameters which control network operation including the actual patterns to be associated and stored.
BUTTONS:
SETUP: The SETUP button prompts the user for
(1) the total number of input-target pairs to be associated, e.g. 2 means: 2 input-target pairs (which is 4 patterns in total);
(2) the learning rate parameter, "r", e.g. 0.25;
(3) the actual input and target patterns with each individual value ("1" or "-1") separated by commas (","), e.g. 1,-1,-1,1
(4) whether the user requires extra information to be displayed during learning. Clicking on "Yes" will slow the learning procedure down, but will also display extra messages explaining individual stages in the learning process.
Note:
Direct entry of patterns and parameters: All the above values (except extra information) may be entered without clicking the SETUP button. Simply place the cursor in the appropriate field, darken any previous values to delete them, and enter the new values in their place. Be sure such parameters as the number of pairs and the learning rate are entered.
LEARN: The LEARN button should be pressed after all appropriate setup values have been entered either by using the SETUP button, or by manually entering the values in the appropriate fields. There is a LEARN button for both Matrix and Graphics display options.
TEST: The TEST button may be clicked on completion of learning after a test pattern has been entered in the "Enter test input" field. There is a TEST button for both the Matrix and Graphics formats.
GO TO: Below the label "Go To" are two buttons to take the user either to the Graphics or Matrix displays.
MENU: The MENU button takes the user back to the main menu.
FIELDS:
TOTAL PATTERNS:
This field displays the total number of pattern pairs to be learned and should be entered by the user in the Setup screen.
LEARNING RATE:
This field displays the learning rate, "r", which should have been entered by the user in the Setup screen.
INPUTS and TARGETS:
These two fields display input and target patterns. The patterns are entered by the user in the Setup screen.
THE GRAPHICS SCREEN
The Graphics Screen displays a fully interconnected neural network with one layer of connection weights. Input units are below with their activation values shown within each unit. Output units are above the inputs with activations shown within each unit. The target units are above the output units. Weights are shown on each connection from input to output unit.
BUTONS:
LEARN: The patterns previously entered in the Setup Screen may be learned by pressing the LEARN button in the Graphics screen.
TEST: Pressing the TEST button takes the user to the Setup Screen where a test input can be entered. Appropriate fields and buttons are flashed to indicate the user's next appropriate action.
SHOW INPUTS: At the completion of learning, the SHOW INPUTS button can display the input-target pairs for comparison to the actual output produced by the network. This is especially useful after testing a partial pattern as test input to check whether the output produced was equal to the output expected from previous learning. Clicking the SHOW INPUTS button once displays pattern pairs. Clicking it again hides the patterns.
FIELDS:
PATTERN NUMBER: The current number of the pattern pair currently being associated by the network is shown in the field called PATTERN No. As each pattern pair is learned, the next new input pattern is placed on the input units and the value in PATTERN No. is increases by one.
CYCLE NUMBER: The value in CYCLE No. increases each time the network completes a cycle. A cycle means one sweep of the input pattern through the weights and comparison of the newly computed outputs to the target pattern. Several cycles may be required before the output pattern equals the target pattern for a given input-target pair.
THE MATRIX SCREEN
The buttons on the Matrix Screen have exactly the same function as on the Graphics Screen (see above). However note that they are positioned differently on the screen. The Matrix display provides a nice representation of the weights and is preferable to the Graphics format for seeing how they are patterned in terms of positive or negative values. This is especially the case when the user explores the "Numbers Simulation" demo that has far too many weights to display in the Graphics format.
SAVING WEIGHTS: The Matrix Screen also provides a facility for saving a set of weights once learned, and loading a set which were saved previously. This allows the user to quickly run the network with a set of weights which may have taken considerable time (!!) to learn initially. Click either the button SAVE WTS or LOAD WTS for this function. Note that these buttons are not replicated in the Graphics Screen so the user should go to the Matrix Screen in order to save or load weights.
The numbers network allows the user to explore the ideas of PATTERN INTERFERENCE and PATTERN GENERALIZATION with a "real world" pattern recognition example. Interference occurs when pattern pairs which are associated together are too similar to each other that the network can not find an appropriate set of weights to store them in a distinct way. There is then "crosstalk" between the patterns and errors may occur in recalling (completing) a pattern.
Concept learning:
Pattern generalization also occurs when different inputs are associated with the same target (e.g. different "chair" visual patterns are each associated with the same pattern for the label "furniture"). If the network is then tested with a new pattern (similar to but different from the initial "chair" patterns), an output may result which actually equals the "furniture" pattern. Thus the pattern of weights in the network can be said to represent the same "concept" of “furniture” which many different chair input patterns all produce.
Learning digits:
We shall explore this idea of generalization by storing numbers (e.g. 1, 2, 3, 4, 5) in the same network. It is recommended, as the example, that you store the first 5 digits (i.e. 1 to 5). Each number is entered into the network as a pattern of "1"s on a 3x5 rectangular grid. This is similar to the pixels of a computer screen. Only the inputs need be entered because the network will associate the input pattern with a copy of itself on the target units. Such a network is called an AUTOASSOCIATOR because it stores a pattern by associating the pattern with itself (i.e. the input and the target is the same pattern).
OPERATION OF THE NUMBERS NETWORK
SETUP: As in the Simulator network, there is a Setup and a Matrix Screen. The Setup Screen provides the basic control over initial parameters, etc. The Matrix Screen provides a matrix display of the network learning process and weights. There is also a Numbers Screen which allows the user to enter input patterns corresponding to digits. The Numbers screen also displays, in digit form, the results of testing the network with partial input patterns.
MATRIX SCREEN
This is also analogous to the Matrix Screen of the Simulator network, and also provides the capability for saving the set of weights once learned. This is very important since it may take 15 minutes or more to learn the weights for a set of five digits! Once learned, then the weights may be saved and loaded for future testing and demonstration of the Numbers Network.
SETUP SCREEN
This has buttons analogous to those in the Setup screen of the basic, 4-node Simulator network.
BUTTONS:
SETUP: This must be clicked FIRST before any others. SETUP allows the user to enter:
a) the total number of patterns to be learned, and
b) the pattern length.
For number patterns, the length MUST equal 15 since there are 15 "pixels" in each number's representation.
ENTER NUMBERS: After the SETUP procedure has been followed, the ENTER NUMBERS button should be clicked. This takes the user to the Numbers Screen.
Each digit to be learned can be entered into the 3x5 grid by placing the cursor over a square and entering a "1". Where there is a space in the pattern representation of a number, do not enter anything in that square. Only enter a "1" where there would be "ink" in a normal representation of a digit.
For example: the representation of the digits "2" and "5" might look like this:
111 1111
1 1
111 1111
1 1
111 1111
Notice that there is a "1" absent wherever there is a blank space in the writing of the digit. You will be asked whether you want to see an example of the digit representation. As usual, you should initially answer with the "Yes" option to display some example digits.
PUT PATTERN: After you have created EACH complete digit pattern, click the button PUT PATTERN. This will store the inputs in the Setup Screen and allow you to then enter another digit into the grid. Again click PUT PATTERN, after each digit is created.
LEARN: When you have entered all the patterns to be learned by clicking PUT PATTERN, you should click the LEARN button which will take you to the Setup Screen. You should then click the LEARN button which will flash. This will take you to the Matrix Screen where the learning process will be displayed.
You can follow the learning process as it happens. However, for these 15 unit-long digit patterns, it takes about two minutes for each digit to be stored through weight modification. This is because of the large number of weights to be updated (15 inputs x 15 outputs = 225 weights in total!). If there are five or so patterns to be stored, it will take about 10-15 minutes for the network to learn all the patterns.
When the learning is complete, you will be asked whether you wish to TEST the network with a new pattern, or one of the old ones. Answering "Yes" to the prompt will take you to the Setup Screen where the TEST INPUT field will flash. You can then either:
1) (This is not recommended initially - see [2] below). Follow the prompt and enter any partial pattern in the TEST field and then click the TEST button. However, it is NOT recommended to follow this option initially since it is difficult to know what the input string actually represents. Instead, do the following:
2) (Recommended for testing digits). Click instead the NUMBERS button which will take you to the Numbers Screen. Here you can enter the complete or partial pattern for a digit previously learned by entering the pattern in the numbers grid and then clicking the TEST button. This will test the network's ability for correct recall given the weights produced during learning. In addition you will be able to observe the pattern completion and pattern blending properties of the network.
TEST: Click the TEST button after entering each test input pattern. This will take you to the Setup Screen where you should click on the TEST button. This takes you to the Matrix Screen where you can observe the output of each test input. After the output has been produced, click the SHOW NUMBER button which will display the output pattern in digit form.
USING THE NUMBERS NETWORK AS A GENERAL PATTERN ASSOCIATOR
The numbers network can be used as a general associative network for associating any input of arbitrary length with any target. Thus the user may experiment with networks of different numbers of units. To save time, input and target patterns may be entered directly into their respective fields in the Setup Screen without pressing the SETUP button, as may the relevant parameters of Total Patterns, Pattern Length, and Learning Rate. Alternatively, these parameters may be entered by clicking the SETUP button first which will prompt the user for each value.